Robotics: Environmental Awareness Through Cognitive Sensor Fusion

In order for autonomous mobile robots to survive in the real world they have to be aware of the environment. The self-assembling micro robots developed in the European project Replicator are destined for such a task. By using several modalities, these robots must be able to detect and recognize interesting objects in the environment. This thesis presents a biologically inspired cognitive sensor fusion architecture to create environmental awareness in these micro robots. This architecture consists of a bi-modal attention module and multimodal sensor fusion. A state of the art visual saliency detection system has been optimized and combined with biologically based sensor fusion methods to obtain a visual-acoustic attention module. For multi-modal sensor fusion a new type of ARTMAP (self-organizing associative memory) called the Multi-directional ARTMAP (MdARTMAP) has been designed. With this MdARTMAP a module for unsupervised on-going learning of sound was developed by clustering states of an echo state network, which processes cochlear filtered audio. Also unsupervised visual object recognition was obtained with this MdARTMAP by clustering and associating salient SIFT keypoint descriptors. Based on thesemodules amulti-modal sensor fusion system was created by hierarchically associating the MdARTMAPs of the different modalities. Experiments conducted in a 3D simulator showed that a simulated robot was able to successfully perform a variety of search tasks with the cognitive sensor fusion architecture.